{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"../../img/ods_stickers.jpg\" />\n",
    "    \n",
    "## [mlcourse.ai](mlcourse.ai) – Open Machine Learning Course \n",
    "### <center> Author: Alexander Nichiporenko, @AlexNich\n",
    "    \n",
    "## <center> Prediction of customers which will buy car insurance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1. Feature and data explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probably, many of us are faced with a situation when a company calls you to buy or buy something. Typical examples:\n",
    "\n",
    "* You use a credit card, and the bank calls you with an offer to issue a loan;*\n",
    "* You bought auto insurance, and the insurance company calls and offers you other types of insurance;\n",
    "* You have been using cellular communication for a long time, and your operator calls you with a proposal to use a new more profitable (oddly enough, more expensive) tariff;\n",
    "* You bought something from an online store, and after a while he calls you to buy another item.\n",
    "* Any situations related to the acquisition of a new service, an additional service, a more expensive service.\n",
    "\n",
    "Usually, in most cases, the client does not agree to such offers, because he simply does not need it. It turns out that ringing the entire customer base is long and inefficient, so companies try to contact only those who are likely to agree to their proposal. How to find such customers? This can be done as follows:\n",
    "\n",
    "* Call a certain random part of clients, record the result;\n",
    "* Find in the remaining customer base of the most similar to those who agreed to the proposed service;\n",
    "* Call these customers, thereby increasing the effectiveness of contacts.\n",
    "\n",
    "We will solve a similar problem. We have a dataset from one bank in the United States. Besides usual services, this bank also provides car insurance services. The bank organizes regular campaigns to attract new clients. The bank has potential customers’ data, and bank’s employees call them for advertising available car insurance options. We are provided with general information about clients (age, job, etc.) as well as more specific information about the current insurance sell campaign (communication, last contact day) and previous campaigns (attributes like previous attempts, outcome). The task is to predict of customers who will buy car insurance or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import libraries\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import cross_val_score, TimeSeriesSplit, GridSearchCV, train_test_split, KFold, learning_curve, validation_curve\n",
    "from sklearn.metrics import accuracy_score,classification_report,f1_score,roc_auc_score,roc_curve,precision_recall_curve\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from xgboost import XGBClassifier\n",
    "plt.rcParams['figure.figsize'] = (20,20)\n",
    "#sns.set(style=\"darkgrid\");\n",
    "%matplotlib inline\n",
    "pd.options.display.max_columns=500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at our dataset. You can download it here: https://www.kaggle.com/kondla/carinsurance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('carInsurance_train.csv',index_col='Id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 4000 customers with 17 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our target variabe - **'CarInsurance'**, which is binary (1/0).\"1\" means that the customer has agreed to the offer, \"0\" means that not.\n",
    "\n",
    "Eighteen features overvies:\n",
    "\n",
    "- **Id** - Unique ID number;\n",
    "- **Age** - Age of the client;\n",
    "- **Job** - Job of the client.  \"admin.\", \"blue-collar\", etc.\n",
    " **Marital** - Marital status of the client  \"divorced\", \"married\", \"single\";\n",
    "- **Education** - Education level of the client \"primary\", \"secondary\", etc.\n",
    "- **Default** - Has credit in default? \"yes\" - 1,\"no\" - 0\n",
    "- **Balance** - Average yearly balance, in USD\n",
    "- **HHInsurance** - Is household insured \"yes\" - 1,\"no\" - 0\n",
    "- **CarLoan** - Has the client a car loan \"yes\" - 1,\"no\" - 0\n",
    "- **Communication** - Contact communication type \"cellular\", \"telephone\", “NA”\n",
    "- **LastContactMonth** -  Month of the last contact \"jan\", \"feb\", etc.\n",
    "- **LastContactDay** - Day of the last contact\n",
    "- **CallStart** - Start time of the last call (HH:MM:SS) 12:43:15\n",
    "- **CallEnd** - End time of the last call (HH:MM:SS) 12:43:15\n",
    "- **NoOfContacts** - Number of contacts performed during this campaign for this client; \n",
    "- **DaysPassed** - Number of days that passed by after the client was last contacted from a previous campaign (numeric; -1 means client was not previously contacted) \n",
    "- **PrevAttempts** - Number of contacts performed before this campaign and for this client \n",
    "- **Outcome** - Outcome of the previous marketing campaign \"failure\", \"other\", \"success\", “NA”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2. Primary data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, examine our data on missing values and outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#devide features in categorical and numerical\n",
    "\n",
    "data['Default']=data['Default'].astype('object')\n",
    "data['HHInsurance']=data['HHInsurance'].astype('object')\n",
    "data['CarLoan']=data['CarLoan'].astype('object')\n",
    "data['LastContactDay']=data['LastContactDay'].astype('object')\n",
    "\n",
    "cat = []\n",
    "num = []\n",
    "for feature in data.drop(columns=['CarInsurance']).columns:\n",
    "    if data[feature].dtype == object:\n",
    "        cat.append(feature)\n",
    "    else:\n",
    "        num.append(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print ('Number of categorical features:',len(cat))\n",
    "print ('Number of numerical features:',len(num))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Job'].isnull()].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Education'].isnull()].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Communication'].isnull()].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Outcome'].isnull()].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see dataset has some missing values: \n",
    "* Job and Education may be missed because customers didn't specify this information;\n",
    "* Communication may be missed because bank didn't fix communication type\n",
    "* Outcome has missing values because some customers haven't been offered anything before, respectively, and there is no outcome;\n",
    "\n",
    "We will fill **NaN's** later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.describe(include = ['object'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some values seem suspicious and may be outliers:\n",
    "\n",
    "* **max Age = 95 years**. Real survivor!\n",
    "* **max Balance = 98 417 USD**, when mean is **1532 USD** and 75% procentile equals to **1619 USD**. May be this man is very rich? It's typical for income distribution.\n",
    "* **min Balance = - 3058 USD**. Maybe this person spent all the credit money and did not return?\n",
    "* **max NoOfContacs = 43**. Did the bank offer so many times insurance within this company to some person? Interestingly, he agreed?\n",
    "* **max DaysPassed = 854**. The bank does not call someone for more than three years?\n",
    "* **max PrevAttempts = 58** when mean is 0.72. \n",
    "\n",
    "Let's look at id with this strange values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Age']==95].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Balance']==98417].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['Balance']==-3058].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['DaysPassed']==854].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['NoOfContacts']==43].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[data['PrevAttempts']==58].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at this data it is impossible to say that there are definitely some errors in the data. Perhaps everything is correct. Later we will visualize the data and decide what to do with suspicious values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the part of customers who bought car insurance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['CarInsurance'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**40%** isn't bad! But I think the bank wants **100%**, so it calls customers several times. In ML terms we can say that our two classes are balanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now examine the influence of our features on the target variable. Firsly, numerical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by=['CarInsurance'])[['Age']].agg([np.mean,np.std,np.min,np.max])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by=['CarInsurance'])[['NoOfContacts']].agg([np.mean,np.std,np.min,np.max])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by=['CarInsurance'])[['DaysPassed']].agg([np.mean,np.std,np.min,np.max])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by=['CarInsurance'])[['PrevAttempts']].agg([np.mean,np.std,np.min,np.max])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by=['CarInsurance'])[['Balance']].agg([np.mean,np.std,np.min,np.max])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the constructed tables, we can see that those customers who agree to insurance in average:\n",
    "\n",
    "* The bank makes more offers with this insurance\n",
    "* Such clients were offered an offer by another bank company on average more than two months ago, for those who did not agree - just over a month\n",
    "* They were more often offered other bank offers\n",
    "* Have a bit more balance\n",
    "* Have less contacts from the bank for other campaigns\n",
    "\n",
    "To confirm these observations we build histograms and boxplots of features futher.\n",
    "Now take a look at categorical and binary features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['Education'],data['Job'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['Marital'],data['Education'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['Default'],data['Job'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['CarLoan'],data['Job'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['CarLoan'],data['HHInsurance'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['Communication'],data['Outcome'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['Communication'],data['LastContactMonth'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(data['LastContactDay'],data['LastContactMonth'],values=data['CarInsurance'],aggfunc='mean',margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these crosstabs we can see:\n",
    "\n",
    "* The way of communication doesn't affect on target variable\n",
    "* Monthly and dayily dependence of campaign\n",
    "* Persons with CarLoan rare agree to the offer\n",
    "* Persons with HHInsurance rare agree to the offer\n",
    "* People who agreed to other offers of the bank more often agree to insurance\n",
    "* Persons with Default rare agree to the offer\n",
    "* Single persons and persons who have tretiary education ofter agree to insurance \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3. Primary visual data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make visualizations of our features and their effect on the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#target distribution\n",
    "sns.countplot(data['CarInsurance'],palette=\"Accent\");\n",
    "plt.title('Target distribution');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#distribution of categorical features\n",
    "\n",
    "plt.figure(figsize=(20,20))\n",
    "for i in range(1,len(cat[:11])):\n",
    "    plt.subplot(4,3,i)\n",
    "    sns.countplot(data[cat[i-1]],palette='Accent')\n",
    "    plt.xticks(rotation=90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that some values of categorical features (**\"Default=1\"** or months) have a small number of examples. In general, such values are usually combined into one group to prevent overfitting, and in the binary case, this column can be deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#target variable versus categorical\n",
    "\n",
    "plt.figure(figsize=(20,20))\n",
    "for i in range(1,len(cat[:11])):\n",
    "    plt.subplot(4,3,i)\n",
    "    sns.barplot(data[cat[i-1]],data['CarInsurance'],palette='Accent')\n",
    "    plt.xticks(rotation=90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusions regarding the dependence of the target variable on categorical features obtained using primary data analysis are confirmed by these visualizations (see **Part 2**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#histograms of numerical features and their scatterplots\n",
    "\n",
    "sns.pairplot(data[num], palette=\"Accent\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_matrix = data[num].corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.heatmap(corr_matrix,cmap=\"Accent\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From scatterplots and heatmap is obviosly that our numerical haven't visible correlations, and the distributions are strongly skewed to the left except for age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#histograms of numerical features and their scatterplot\n",
    "\n",
    "sns.pairplot(data[num + ['CarInsurance']],hue='CarInsurance',palette=\"Accent\",diag_kind='kde');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#boxplots depending on the target variable\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "for i in range(1,len(num)+1):\n",
    "    plt.subplot(2,3,i)\n",
    "    sns.boxplot(data=data, x=data['CarInsurance'],y=data[num[i-1]],palette=\"Accent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#graphs depending on the target variable with a limit of 0.975 quantile for better visibility\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "for i in range(1,len(num)+1):\n",
    "    plt.subplot(2,3,i)\n",
    "    sns.boxplot(data=data, x=data['CarInsurance'],y=data[data[num[i-1]]<data[num[i-1]].quantile(0.975)][num[i-1]],palette=\"Accent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, all conclusions and influinces also agree with what was obtained as a result of the analysis in **Part 2**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4. Insights and found dependencies "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's summarize, what patterns were discovered:\n",
    "\n",
    "* Tretiary education rises chances to accept insurance offer, these persons may be more responsible and prudent;\n",
    "* Persons without car's loan and house insurance more loyal to car insurance offer, but it looks a little bit strange\n",
    "* Persons who applied other bank's offers is more loyal to car insurance offer;\n",
    "* If bank offers insurance many times, then it’s more likely that the customer will agree to it;\n",
    "* Persons who were last called in March, September, October and December very likely agreed to the offer. This may be due to seasonality of car sales. Usually insure new cars, and in these months dealers make good discounts on cars;\n",
    "* Single persons often buy car insurance, may be they have extra money to this service. Married persons spend money to other things;\n",
    "* People which buy car insurance have a little bit balance\n",
    "* People to whom the bank has never offered its other services are less likely to agree to car insurance. These are new customers with whom the bank has not yet built a relationship.\n",
    "* Students more often buy car insurance. I think they are newbies in driving so they need insurance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5. Metrics selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we have data on **4 000** clients, and this is ** 10% **  of the entire database. If we assume that the effectiveness of calls for the remaining customers will be about the same, then the bank is interested in calling all customers who agree to insurance for a smaller number of calls or the number of calls that the bank can make. Thus, it would be possible to choose metric **recall@topK%**. As the metric, **K** in this case would be equal to about **50%**. In general, the strategy and capabilities of the bank may vary, so you need to have a universal classifier. In general, the strategy and capabilities of the bank may vary, so you need to have a universal classifier. In this case, the universal metric of work quality classifiers is **ROC-AUC**. We will use it. In this case we can choose threshold and calculate **K** (what part of customers have higher probability) and **recall**@**topK%**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 6. Model selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our dataset there are numerical features with very large values, but they do not contradict anything, so we will leave them unchanged, and use **XGBoost** as a prediction model, which is not afraid to such outliers. Also this algorithm has the best perfomance in most of tasks. Also this task doesn't connect with financial risks, so we can make a \"blackbox\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parts 7-9.Data preprocessing. Cross-validation and adjustment of model hyperparameters. Creation of new features and description of this process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, let's fill **NaN's**. We assumed that some people did not fill in the fields of **Education** and **Job** for any reason, so instead of passes, we put \"unknown\", we will do the same with the type of communication. Missing values in feature **Outcome** we will fill with \"no_outcome\". In general, we simply denote the missing values as another category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['Education'].fillna('unknown',inplace=True)\n",
    "data['Job'].fillna('unknown',inplace=True)\n",
    "data['Communication'].fillna('unknown',inplace=True)\n",
    "data['Outcome'].fillna('no_outcome',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For coding our categorical features we will use common method **OHE** using **pd.get_dummies**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.concat([data.drop(columns=['Job','Marital','Education','Communication','LastContactMonth','Outcome']),pd.get_dummies(data[['Job','Marital','Education','Communication','LastContactMonth','Outcome']])],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first, we will not use \"CallStart\" and \"CallEnd\" features, because we need to work on them and make new features from them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get X and y\n",
    "\n",
    "X = data.drop(columns=['CallStart','CallEnd','CarInsurance'])\n",
    "X=X.astype('float')\n",
    "y = data['CarInsurance']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Devide our dataset by train and valid parts. We will use **25%** for validation. Because we have balanced classification task we won't use stratified splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, random_state=33)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#part of class \"1\" \n",
    "y_train.mean(), y_valid.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check quality of XGBoost via CV with 5 shuffling folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = XGBClassifier(random_state=33, n_jobs=4)\n",
    "kf = KFold(random_state=33,n_splits=5,shuffle=True)\n",
    "print ('Mean ROC-AUC CV score:', np.mean(cross_val_score(xgb, X_train, y_train, scoring='roc_auc',cv=kf)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now we try to add some extra features. We could add the day of the week, but unfortunately we don’t know the year when the calls were made. Therefore, we will work with the signs associated with the call time:\n",
    "* start hour of call\n",
    "* start minute of call\n",
    "* call duration in seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['CallDuration']=pd.to_datetime(data['CallEnd'])-pd.to_datetime(data['CallStart'])\n",
    "data['CallDuration']=data['CallDuration'].dt.total_seconds()\n",
    "data['CallHourStart']=pd.to_datetime(data['CallStart']).apply(lambda t: t.hour)\n",
    "data['CallMinStart']=pd.to_datetime(data['CallStart']).apply(lambda t: t.minute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at our new features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "sns.boxplot(data=data, x=data['CarInsurance'],y=data['CallDuration'],palette=\"Accent\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "sns.barplot(data['CallHourStart'],data['CarInsurance'],palette='Accent');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,6))\n",
    "sns.barplot(data['CallMinStart'],data['CarInsurance'],palette='Accent');\n",
    "plt.xticks(rotation=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**\"CallDurations\"** is very usefull feature, longer calls lead to the purchase of insurance. Other features don't seem so much usefull, but we will try all of them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get X and y\n",
    "\n",
    "X = data.drop(columns=['CallStart','CallEnd','CarInsurance'])\n",
    "X=X.astype('float')\n",
    "y = data['CarInsurance']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, random_state=33)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#part of class \"1\" \n",
    "y_train.mean(), y_valid.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check quality again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = XGBClassifier(random_state=33, n_jobs=4)\n",
    "kf = KFold(random_state=33,n_splits=5,shuffle=True)\n",
    "print ('Mean ROC-AUC CV score:', np.mean(cross_val_score(xgb, X_train, y_train, scoring='roc_auc',cv=kf)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoah! New features gave a noticeable increase in quality! Let's tune hyperparameters via GridSearchCV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "parameters = {'n_estimators':[40, 50, 60, 80, 100, 150, 200, 300], 'max_depth':[3, 4, 5, 6, 7, 8], 'min_child_weight': [1,3,5,7,9]}\n",
    "xgb = XGBClassifier(random_state=33, n_jobs=4)\n",
    "clf = GridSearchCV(xgb, parameters, scoring='roc_auc', cv=kf)\n",
    "clf.fit(X_train, y_train)\n",
    "print('Best parameters: ', clf.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now check new hyperparameters via our CV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb = XGBClassifier(random_state=33, n_jobs=4,max_depth=4, min_child_weight=1, n_estimators=200)\n",
    "kf = KFold(random_state=33,n_splits=5,shuffle=True)\n",
    "print ('Mean ROC-AUC CV score:', np.mean(cross_val_score(xgb, X_train, y_train, scoring='roc_auc',cv=kf)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is now higher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 10. Plotting training and validation curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_with_std(x, data, **kwargs):\n",
    "        mu, std = data.mean(1), data.std(1)\n",
    "        lines = plt.plot(x, mu, '-', **kwargs)\n",
    "        plt.fill_between(x, mu - std, mu + std, edgecolor='none',\n",
    "                         facecolor=lines[0].get_color(), alpha=0.2)\n",
    "        \n",
    "def plot_learning_curve(clf, X, y, scoring, cv=5):\n",
    " \n",
    "    train_sizes = np.linspace(0.05, 1, 20)\n",
    "   \n",
    "    n_train, val_train, val_test = learning_curve(clf, X=X, y=y, train_sizes=train_sizes, cv=cv,scoring=scoring)\n",
    "    plot_with_std(n_train, val_train, label='training scores', c='green')\n",
    "    plot_with_std(n_train, val_test, label='validation scores', c='red')\n",
    "    plt.xlabel('Training Set Size'); plt.ylabel(scoring)\n",
    "    plt.legend()\n",
    "\n",
    "def plot_validation_curve(clf, X, y, cv_param_name, \n",
    "                          cv_param_values, scoring):\n",
    "\n",
    "    val_train, val_test = validation_curve(clf, X, y, cv_param_name, cv_param_values, cv=5, scoring=scoring)\n",
    "    plot_with_std(cv_param_values, val_train, \n",
    "                  label='training scores', c='green')\n",
    "    plot_with_std(cv_param_values, val_test, \n",
    "                  label='validation scores', c='red')\n",
    "    plt.xlabel(cv_param_name); plt.ylabel(scoring)\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learning curve\n",
    "plt.figure(figsize=(12,6))\n",
    "plot_learning_curve(xgb,X_train, y_train, scoring='roc_auc', cv=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering the learning curve, we can say that adding data could improve the quality of the models, because With the addition of new data, the quality of validation is increasing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# validation curve\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "max_depth = [3, 4, 5, 6, 7, 8]\n",
    "plot_validation_curve(XGBClassifier(random_state=33, n_jobs=4, min_child_weight=1, n_estimators=200), X_train, y_train, \n",
    "                    cv_param_name='max_depth', \n",
    "                    cv_param_values=max_depth,\n",
    "                    scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# validation curve\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "n_estimators = [40, 50, 60, 80, 100, 150, 200, 300]\n",
    "plot_validation_curve(XGBClassifier(random_state=33, n_jobs=4, min_child_weight=1, n_estimators=200), X_train, y_train, \n",
    "                    cv_param_name='n_estimators', \n",
    "                    cv_param_values=n_estimators,\n",
    "                    scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# validation curve\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "min_child_weight = [1,3,5,7,9]\n",
    "plot_validation_curve(XGBClassifier(random_state=33, n_jobs=4, min_child_weight=1, n_estimators=200), X_train, y_train, \n",
    "                    cv_param_name='min_child_weight', \n",
    "                    cv_param_values=min_child_weight,\n",
    "                    scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation curves show that the result on the CV is much lower than on the train. This indicates a overfitting of the model. For such a small dataset and boosting is a common thing. To reduce the degree of overfitting, you can try to reduce the complexity of the model and increase the parameters responsible for regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 11. Prediction for test or hold-out samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use our XGBoost to predict probabilities to our X_valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.fit(X_train,y_train)\n",
    "y_pred_valid=xgb.predict_proba(X_valid)[:,1]\n",
    "\n",
    "print ('ROC-AUC score of X_valid:', roc_auc_score(y_valid, y_pred_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obtained score a bit higher then on CV, it means that our CV is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#look at most important features\n",
    "xgboost.plot_importance(xgb,max_num_features=15,importance_type='gain');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 12. Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project we made a model with a good quality **~0.92 ROC-AUC**, so the bank can use it to find customers who are most likely to buy car insurance, depending on the capabilities and policies of the bank."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
